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Executive Summary of Research 


The user interface is the component of a software system that connects two very complex 
system: humans and computers. Each of these two systems impose certain requirements on the final 
product. The user is the judge of the usability and utility of the system; the computer software and 
hardware are the tools with which the interface is constructed. Mistakes are sometimes made in 
designing and developing user interfaces because the designers and developers have limited knowl- 
edge about human performance (e.g., problem solving, decision making, planning, and reasoning). 
Even those trained in user interface design make mistakes because they are unable to address all of the 
known requirements and constraints on design. Evaluation of the user interface is therefore a critical 
phase of the user interface development process. Evaluation should not be considered the final phase 
of design; but it should be part of an iterative design cycle with the output of evaluation being feed 
back into design. 

The goal of this research was to develop a set of computer-based tools for objectively evalu- 
ating graphical user interfaces. The research was organized into three phases. The first phase resulted 
in the development of an embedded evaluation tool which evaluates the usability of a graphical user 
interface based on a user’s performance. An expert system to assist in the design and evaluation of 
user interfaces based upon rules and guidelines was developed during the second phase. During the 
final phase of the research an automatic layout tool to be used in the initial design of graphical inter- 
faces was developed. The research was coordinated with NASA Marshall Space Flight Center’s 
Mission Operations Laboratory’s efforts in developing onboard payload display specifications for the 
Space Station. 
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Summary of First Year Funding (1/25/94 - 1/24/95): 


Research Objective. The first objective of the research, to develop an embedded evalua- 
tion capability to evaluate the adequacy of a graphical user interface based on a user’s performance, 
was accomplished. A Graphical User Interface Evaluation Tool (GUIET) was developed to assist 
designers in the process of formative evaluation which is conducted through usability studies. Given 
a functional prototype and tasks that can be accomplished on that prototype, the designer observes 
how users interact with the prototype to accomplish those tasks in order to identify improvements for 
the next design iteration. Evaluation of the interaction is measured in terms of specific parameters 
including: time to learn to use the system, speed of task performance, rates and types of errors made 
by users, retention over time, and subjective satisfaction. Analysis of this information assists in re- 
design of the system. 

The conceptual model of a designer is a description of the system and how the user should 
interact with it in terms of completing a set of tasks. The user’s mental model is a model formed by 
the user of how the system works, and it guides the user’s actions. Most interaction problems occur 
when the user has an inaccurate model of the system or when the user’s model of a system does not 
correspond with the designer’s conceptual model of the system. The evaluation approach utilized by 
GUIET evaluates the user’s mental model of the system against the designer’s conceptual model. 

Research Accomplishments. A rule-based evaluation approach, implemented using 
CLIPS, was used to develop the conceptual model. The model outlines the specific actions that the 
user must take in order to complete a task. Task correctness is evaluated based mainly on three eval- 
uation criteria: the existence of certain actions, the sequencing of actions, and the time associated with 
the completions of the actions or task. Throughout the evaluation process, user actions are continu- 
ously associated with a set of possibly changing goals. Once a goal has been identified, the user’s 
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action in response to that goal are evaluated to determine if a user has performed a task correctly. User 
performance can be classified into three levels for most tasks - expert, intermediate, and novice. There 
may also be no response to a task. The purpose of the evaluation is not to classify or evaluate users, 
but to evaluate the interface. The classification of users into categories is done to identify the level at 
which the users are interacting with the system. The goal is to have most if not all interactions at what 
the designer would consider the expert level. If users are not interacting at this level, it is the interface 
which must be enhanced to improve user performance. 

An early prototype of the system was demonstrated and integrated into the Human Computer 
Interface Prototyping Environment of the Crew Systems Engineering Branch of the Mission Opera- 
tions Laboratory of NASA Marshall Space Flight Center. The purpose of this environment was to 
assist in the design and evaluation of the Space Station’s on-board payload displays. The Human- 
Computer Interface Prototyping Environment with Embedded Evaluation capability is designed to 
allow a developer to create a rapid prototype of a system and to specify correct procedures for oper- 
ating the system. The first component of the architecture is the Graphical User Interface (GUI) 
development tool. This tool allows the designer to graphically create the interface of the system and 
specify a data source for each object within the display. The simulator tool provides the capability to 
create a low-fidelity simulation of the system to drive the interface. GUIET allows the designer to 
specify which actions need to be taken to complete a task, what actions should be taken in response to 
certain events (e.g., malfunctions), and the time frames in which these actions should be taken. 

The goal of GUIET within the environment is to provide for automatic evaluation of user 
actions. Using GUIET, the process of formative evaluation has more flexibility and takes less time for 
analysis. The main benefits are that the evaluation of most of the participant’s actions are automated, 
and the evaluation is performed at runtime by an expert system. The knowledge base of the system 
contains the designer’s conceptual model, of how he/she thinks the user should interact with the pro- 
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totyped system. Because the knowledge base is not hard coded into the application, it can be dynam- 
ically changed according to the needs of the evaluator. This provides the flexibility to evaluate 
different interfaces with the same evaluation criteria or one interface with different evaluation criteria. 
This design saves time because the data is automatically collected and analyzed based on the rule 
based conceptual model. If a new interface is prototyped, the only change that needs to be made with 
GUIET is changing the knowledge base. 

Conclusions. The rule-based design of conceptual models enables the iterative process of 
design and evaluation to proceed more efficiently. A designer can specify user performance criteria 
prior to evaluation, and the system can automatically evaluate the human computer interaction based 
on the criteria previously specified. In order to evaluate the system, a study was planned at NASA 
Marshall Space Flight Center (MSFC) which would evaluate user performance (using the rule based 
system developed) using a good interface and a bad interface. The hypothesis was that the good 
interface would produce more user responses at the expert level, and the bad interface would produce 
less acceptable responses. Data collected though GUIET provided the basis for identification of good 
user interfaces and bad user interfaces, and improvement suggestions for the bad interfaces. 
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Summary of Second Year Funding (1/25/95 - 5/26/96): 

Research Objectives. There were two objectives for the second year of work. The first 
objective was to extend the first year’s work to include a training component. During the first year a 
Graphical User Interface Evaluation Tool was developed to assist designers in the process of forma- 
tive evaluation. This approach was extended to provide the user training on the designer’s conceptual 
model. The second objective was the development of an expert system to assist in the design and 
evaluation of user interfaces based upon rules and guidelines. The knowledge base of this expert 
system consists of principles, guidelines, standards, and rules which represent the accumulated knowl- 
edge about user interfaces. 

Research Accomplishments. A Generic (Rule-Based) Training System (GETS) was 
developed to provide the user information on the designer’s conceptual model of a system. The sys- 
tem was developed in C and Motif. It is a rule based system that uses an expert system tool, CLIPS, 
to control the execution of rules that describe the application. GETS is composed of three modules: 
the application module, the training system module, and the UAN-Rule translator module. The ap- 
plication system is the external system that the user will be trained to use. In order to connect the 
application system to the other modules, two requirements must be fulfilled. The first is that a com- 
plete description of the actions that the user must perform on the interface to complete a task must be 
provided in the User Action Notation (UAN). The UAN provides an easy, understandable and com- 
plete way of describing user actions, such as moving the cursor to an object, or clicking a button. The 
second requirement is that the actions of the user with the interface must be sent through sockets to the 
training system. The UAN-Rule translator is in charge of creating the rules the expert system will use. 
It creates these rules from an input file that contains the UAN description of all tasks that can be 
performed using the application system. The UAN-Rule translator also adds to the output file con- 
structs to be used in the control of the training process. The training system module is in charge of 
monitoring the steps the user is performing on the application system and offering assistance when- 
ever the user requires or requests it. 

The Expert Advice System for Graphical User Interface Design and Evaluation (EASGUIDE) 
was also developed. It is a Motif based graphical user interface that contains a database, an expert 
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system, and an evaluation tool. The system was implemented in C and uses Oracle and embedded 
SQL for the database component, and CLIPS for the expert system component. The user is not re- 
quired to have any knowledge of the formulation of database queries or the execution of rules within 
an expert system in order to utilize the system. General information on the design and evaluation of 
Graphical User Interfaces (GUIs) is stored in the database. The user might also request design advice, 
such as the best background color or the most appropriate foreground color to be used with a particular 
background. Other types of information on widgets is also available, including the most appropriate 
widget to use to represent a particular object. The design advice is stored within the expert system. 
EASGUIDE provides the user with information on a variety of topics through direct manipulation. It 
presents information on when a particular interface object should be used, and the advantages and 
disadvantages of various interaction styles. EASGUIDE automatically formulates the queries neces- 
sary to retrieve informauon from the database and displays it to the user. The knowledge base along 
with the expert system provide design advice on topics including appropriate background and fore- 
ground colors, fonts, when to use popup menus versus pull down menus, and how to color code 
different statuses. 

The Generic Training System (GETS) was evaluated using the Hopkins Ultraviolet Telescope 
(HUT). A graphical interface for HUT was been developed using Motif, and a simulation of the 
system in order to provide actual feedback was developed using CLIPS. The interface was described 
using the User Action Notation. The systems were demonstrated and integrated into the Human Com- 
puter Interface Prototyping Environment. 

Conclusions. The suite of tools which have been developed so far includes: a Graphical 
User Interface Evaluation Tool (GUIET) for evaluating dynamic interaction between the user and the 
system; a Generic Training System (GETS) to assist in training users in the use of a system; and an 
Expert Advice System for Graphical User Interface Design and Evaluation (EASGUIDE) to provide 
information and advice about design. These tools provide valuable assistance in the process of design 
and evaluation of graphical user interfaces. 
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Summary of Third Year Funding (5/27/96 - 5/26/97): 

Research Objectives. The final phase of the research involved the development of a tool 
to automatically layout graphical interfaces based upon task analysis. The tool accepts a description 
of the tasks to be accomplished using the interface. The system then decides the objects which are 
needed to accomplish the tasks. The objects will are then placed on the screen using a graph layout 
method. In order to develop this tool, two objectives were identified. The first objective was to create 
an automatic component generation tool that speeds interface code development by giving the 
interface designer the ability to generate code during the design process using user task descriptions. 
In addition to generating the code for each component widget, the system would allow the user to view 
the code, the reasons for choosing a particular widget type, and the widget. 

The second objective was the physical layout of the components. The Component Generation 
tool would be used to identify the widgets and a graph layout algorithm would be used to decide the 
placement of widgets. An algorithm had to be developed to convert the output of the CGP program to 
the input specification for the graph program. 

Research Accomplishments. We have constructed an interface Component Generation 
Program (CGP) that allows user interface designers to create a well designed user interface by 
automating the selection of interface objects, or widgets, based on task analysis. The tool allows the 
user to specify a UAN task description file. This file is parsed and primitive actions are identified. 
The system then uses rules to determine what object should be generated in order to allow a paticular 
user action to be carried out. The system incorporates general human usability factors and design 
guidelines into the generation rules. The generated interface objects are also based on task analysis 
rules governing the precedence and sequence within the UAN. The tool next generates C/Motif code 
for each widget. The user can view the code segment for each widget, view the widget that was 
created by the code, or obtain an explanation as to why a particular widget type was chosen during the 
creation process. 

For the layout component of the tool a graph layout algorithm (CG) was investigated. The 
input used was the task analysis notation, UAN, used by the component generation tool. A new 
algorithm, the Task Specified Layout (TSL) algorithm, was developed to extract interface object and 
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object relationship information from the UAN. The results of this research showed that the User 
Action Notation provides significant amounts of information regarding the potential interface objects 
needed, and the relationships between objects of an interface. The TSL algorithm demonstrated the 
concept of grouping siblings into interface objects. Also, the parent-child relationship is a valid ap- 
proach for identifying sibling elements within a given UAN. Even though there will be exceptions to 
the rule of grouping siblings into a single widget, sibling identification is still needed, so that they can 
be grouped in the same window or area of a given interface. At first glance, it is clear that the CG 
produced layouts of this research lacked aesthetic qualities, such as symmetry, and balance. This was 
due not only to the layout, but also the widgets that were chosen by the widget recognition tool. 
However, a closer review of the results shows that CG did produce good results in the areas of orga- 
nizing related widgets close to each other, and presenting a physical organization that assimilates the 
sequence, or flow of a given user task. 

Conclusions. The fundamental concept of using the automatic code generation tool, the CG 
graph layout tool, and the TSL algorithm to build an interface from a User Action Notation input 
specification was successfully demonstrated here. However, for this concept to produce a useful 
interface, further work is needed in the area of layout. The Component Generation tool has been 
demonstrated and integrated into the Human Computer Interface Prototyping Environment. We plan 
to demonstrate the layout tool in the Fall quarter of 1997. 

Software Tools which Resulted: 

A Graphical User Interface Evaluation Tool (GUIET) 

A Generic Training System (GETS) 

An Expert Advice System for Graphical User Interface Design and Evaluation (EASGUIDE) 
An Interface Component Generation Tool (CGP) 

A Graphical Interface Layout Algorithm 
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Publications which Resulted: 


Several refereed conference papers. Master’s theses, and Master’s projects have been complet- 
ed and presented. They include the following: 

Bester, Terri (1995). Expert Advice System on Graphical User Interface Design and 
Evaluation. Unpublished Master’s Thesis, Auburn University, Auburn, Alabama. Di- 
rected by Dr. Loretta A. Moore. 

Blocker, Susan (1996). The Automatic Generation of User Interface Components 
Based on Task Analysis. Unpublished Master’s Thesis, Auburn University, Auburn, 
Alabama. Directed by Dr. Loretta A. Moore. 

Holmquist, Selma (1995). GETS: A Generic (Rule-Based) Training System. Unpub- 
lished Master’s Thesis, Auburn University, Auburn, Alabama. Directed by Dr. Loretta 
A. Moore. 

Holmquist. S. & Moore, L. A. (1997). "The Generation of Rules for Intelligent Help 
Based on Task Analysis," Proceedings of the 35th Annual ACM Southeast Conference, 
Murfreesboro, Tennessee, April, 1997. 

Link, David (1997). Can Graph Layout Algorithms be Used for Graphical Interface 
Layout? Unpublished Master’s Project Report, Auburn University, Auburn, Alabama. 
Directed by Dr. Loretta A. Moore. 

Moore, L. A., Chang, K., Hale, J., Bester, T., Rix, T., & Wang, Y. (1994). "Rule- 
Based Design of Conceptual Models for Formative Evaluation," Proceedings of the 
Third Conference on CLIPS, Houston, Texas, September, 1994. 

Moore, L. A., Price, S., & Hale, J. (1994). "Simulation in a Dynamic Prototyping 
Environment: Petri Nets or Rules?" Proceedings of the Third Conference on CLIPS, 
Houston, Texas, September, 1994. 

Owen, William (1995). A Framework for the Creation of Conceptual Models. Un- 
published Master’s Thesis, Auburn University, Auburn, Alabama. Directed by Dr. 

Loretta A. Moore. 

Price, Shannon (1994). An Improved Simulation Architecture. Unpublished Master’s 
Project Report, Auburn University, Auburn, Alabama. Directed by Dr. Loretta A. 

Moore. 

Rix, Thomas (1994). GUIET: Graphical User Interface Evaluation Tool. Unpublished 
Master’s Project Report, Auburn University, Auburn, Alabama. Directed by Dr. Lor- 
etta A. Moore. 

Several other conference papers and journal articles are being prepared based upon the last 
year of the research. 
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Summary Data on Project Personnel: 

Students supported through the grant include both undergraduate and graduate students. Un- 
dergraduate students supported include Trina Knight (African-American Female), Roderick Chaney 
(African-American Male), Pamela Williams (African-American Female), Danielle Giglio (White Fe- 
male), Jordan Cochran (White Male), Timothy Zane (White Male), Leslie Reyes (Hispanic Female), 
and Alicia Thomas (African-American Female). Graduate students supported include Hope Gray 
(African-American Female), Terri Bester (African-American Female), David Daniels (African- 
American Male), Susan Blocker (White Female), Shannon Price (White Male), and Thomas Rix 
(White Male). In addition to the students supported, Ms. Selma Holmquist (Hispanic Female), a stu- 
dent with support from another source, worked on the project. Four Master’s theses and three 
Master’s projects have been completed and presented during the grant period. All eight undergraduate 
students have completed their Bachelor’s degrees. 

During the granting period. Dr. Moore received both tenure and promotion to associate 
professor. 

Conclusions: 

The capability to objectively evaluate graphical user interfaces can significantly enhance the 
quality and functionality of the interfaces while at the same time reduce development and training 
costs. The research which has been completed provides a suite of computer-based tools for objec- 
tively evaluating graphical user interfaces which will be valuable in supporting and evaluating 
program and project user interface development. Most of our collaboration has been with the Mission 
Operation Laboratory’s efforts in developing onboard, graphical Payload Displays for the Space Sta- 
tion; however, this tool could also prove valuable to other NASA projects. We have demonstrated 
most of the tools at Marshall Space Flight Center, and installed most prototypes on their system. 

In addition to the technical objectives, NASA’s three program objectives were met: 1) to 
identify early in their career outstanding underrepresented minority engineering, physical and life 
sciences faculty members to contribute to NASA research objectives; 2) to provide support for the 
investigators to move toward NASA mainstream research; and 3) to develop a pool of underrepre- 
sented minority graduates, who are U.S. citizens, with research experience in NASA related fields 
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Transition to NASA Mainstream Research: 


As a result of the experience gained through the funding from this grant. Dr. Moore was selected as a 
participant in the NASA/ASEE Summer Faculty Fellow Program at Kennedy Space Center. (Prior to 
the beginning of this grant Dr. Moore has participated in the NASA/ASEE Summer Faculty Fellow 
Program at Marshall Space Flight Center working with Mr. Joe Hale, Technical Contact.) Dr. 
Moore’s research experience in the development of tools and environments for the evaluation of 
graphical user interfaces was utilized in the Payload Directorate. There she worked with the Portable 
Data Collection Team in the design and evaluation of the PDC system, which is designed to develop 
a paperless work procedure system. As a result of her work there and prior experience with NASA 
grants and contracts, she was invited to submit two proposals. The first for the application of a user- 
centered design process to the development of a Work Authorization Document (WAD) Creation 
Tool, and the second for the development of a KSC Usability Laboratory. 
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